{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. \n",
    "\n",
    "In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "Dropout[1]是一种神经网络正则化的技术，它通过在前向传播中将一些输出激活随机设置为零。\n",
    "\n",
    "在这个练习中，您将实现一个dropout层，并修改您的全连接网络，以选择使用dropout。\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "run the following from the cs231n directory and try again:\n",
      "python setup.py build_ext --inplace\n",
      "You may also need to restart your iPython kernel\n"
     ]
    }
   ],
   "source": [
    "# As usual, a bit of setup\n",
    "from __future__ import print_function\n",
    "import time\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from cs231n.classifiers.fc_net import *\n",
    "from cs231n.data_utils import get_CIFAR10_data\n",
    "from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array\n",
    "from cs231n.solver import Solver\n",
    "\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0)  # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.rcParams['image.cmap'] = 'gray'\n",
    "\n",
    "# for auto-reloading external modules\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "\n",
    "def rel_error(x, y):\n",
    "    \"\"\" returns relative error \"\"\"\n",
    "    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "pdf-ignore"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train:  (49000, 3, 32, 32)\n",
      "y_train:  (49000,)\n",
      "X_val:  (1000, 3, 32, 32)\n",
      "y_val:  (1000,)\n",
      "X_test:  (1000, 3, 32, 32)\n",
      "y_test:  (1000,)\n"
     ]
    }
   ],
   "source": [
    "# Load the (preprocessed) CIFAR10 data.\n",
    "\n",
    "data = get_CIFAR10_data()\n",
    "for k, v in data.items():\n",
    "    print('%s: ' % k, v.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout forward pass\n",
    "In the file `cs231n/layers.py`, implement the forward pass for dropout. \n",
    "\n",
    "Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Once you have done so, run the cell below to test your implementation.\n",
    "\n",
    "由于dropout在训练和测试期间的行为不同，请确保实现两种模式的操作。\n",
    "\n",
    "完成之后，运行下面的单元以测试您的实现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running tests with p =  0.25\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  10.014059116977283\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.749784\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.4\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.977917658761159\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.600796\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n",
      "Running tests with p =  0.7\n",
      "Mean of input:  10.000207878477502\n",
      "Mean of train-time output:  9.987811912159426\n",
      "Mean of test-time output:  10.000207878477502\n",
      "Fraction of train-time output set to zero:  0.30074\n",
      "Fraction of test-time output set to zero:  0.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(500, 500) + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "    out, _ = dropout_forward(x, {'mode': 'train', 'p': p})\n",
    "    out_test, _ = dropout_forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "    print('Running tests with p = ', p)\n",
    "    print('Mean of input: ', x.mean())\n",
    "    print('Mean of train-time output: ', out.mean())\n",
    "    print('Mean of test-time output: ', out_test.mean())\n",
    "    print('Fraction of train-time output set to zero: ', (out == 0).mean())\n",
    "    print('Fraction of test-time output set to zero: ', (out_test == 0).mean())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dropout backward pass\n",
    "In the file `cs231n/layers.py`, implement the backward pass for dropout. \n",
    "\n",
    "After doing so, run the following cell to numerically gradient-check your implementation.\n",
    "\n",
    "这样做之后，运行以下单元格来进行数值梯度检查。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dx relative error:  5.44560814873387e-11\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "x = np.random.randn(10, 10) + 10\n",
    "dout = np.random.randn(*x.shape)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 123}\n",
    "out, cache = dropout_forward(x, dropout_param)\n",
    "dx = dropout_backward(dout, cache)\n",
    "dx_num = eval_numerical_gradient_array(lambda xx: dropout_forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 1:\n",
    "What happens if we do not divide the values being passed through inverse dropout by `p` in the dropout layer? Why does that happen?\n",
    "\n",
    "如果我们不将通过反向dropout传递的值除以dropout层中的`p`会发生什么?为什么会这样?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fully-connected nets with Dropout\n",
    "In the file `cs231n/classifiers/fc_net.py`, modify your implementation to use dropout. \n",
    "\n",
    "Specifically, if the constructor of the network receives a value that is not 1 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity. \n",
    "\n",
    "After doing so, run the following to numerically gradient-check your implementation.\n",
    "\n",
    "具体来说，如果网络的构造函数收到的`dropout`参数的值不是1，那么网络应该在每个ReLU非线性之后立即添加一个dropout层。\n",
    "\n",
    "这样做之后，运行以下代码来进行数值梯度检查。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running check with dropout =  1\n",
      "Initial loss:  2.3004790897684924\n",
      "W1 relative error: 1.48e-07\n",
      "W2 relative error: 2.21e-05\n",
      "W3 relative error: 3.53e-07\n",
      "b1 relative error: 5.38e-09\n",
      "b2 relative error: 2.09e-09\n",
      "b3 relative error: 5.80e-11\n",
      "\n",
      "Running check with dropout =  0.75\n",
      "Initial loss:  2.302371489704412\n",
      "W1 relative error: 1.90e-07\n",
      "W2 relative error: 4.76e-06\n",
      "W3 relative error: 2.60e-08\n",
      "b1 relative error: 4.73e-09\n",
      "b2 relative error: 1.82e-09\n",
      "b3 relative error: 1.70e-10\n",
      "\n",
      "Running check with dropout =  0.5\n",
      "Initial loss:  2.3042759220785896\n",
      "W1 relative error: 3.11e-07\n",
      "W2 relative error: 1.84e-08\n",
      "W3 relative error: 5.35e-08\n",
      "b1 relative error: 2.58e-08\n",
      "b2 relative error: 2.99e-09\n",
      "b3 relative error: 1.13e-10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(231)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = np.random.randn(N, D)\n",
    "y = np.random.randint(C, size=(N,))\n",
    "\n",
    "for dropout in [1, 0.75, 0.5]:\n",
    "    print('Running check with dropout = ', dropout)\n",
    "    model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dtype=np.float64,\n",
    "                            dropout=dropout, seed=123)\n",
    "\n",
    "    loss, grads = model.loss(X, y)\n",
    "    print('Initial loss: ', loss)\n",
    "    \n",
    "    # 相对误差应在e-6左右或更小;\n",
    "    # 请注意，如果对于dropout=1你有 W2 error在e-5的数量级，这是没有问题的。\n",
    "    # Relative errors should be around e-6 or less; Note that it's fine\n",
    "    # if for dropout=1 you have W2 error be on the order of e-5.\n",
    "    for name in sorted(grads):\n",
    "        f = lambda _: model.loss(X, y)[0]\n",
    "        grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)\n",
    "        print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regularization experiment\n",
    "As an experiment, we will train a pair of two-layer networks on 500 training examples: one will use no dropout, and one will use a keep probability of 0.25. \n",
    "\n",
    "We will then visualize the training and validation accuracies of the two networks over time.\n",
    "\n",
    "作为一个实验，我们将在500个训练样本上训练一对两层网络:一个不使用dropout，一个使用保持概率0.25。\n",
    "\n",
    "然后，我们将随时间可视化两个网络的训练和验证准确性。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "(Iteration 1 / 125) loss: 7.856643\n",
      "(Epoch 0 / 25) train acc: 0.260000; val_acc: 0.184000\n",
      "(Epoch 1 / 25) train acc: 0.416000; val_acc: 0.258000\n",
      "(Epoch 2 / 25) train acc: 0.482000; val_acc: 0.276000\n",
      "(Epoch 3 / 25) train acc: 0.532000; val_acc: 0.277000\n",
      "(Epoch 4 / 25) train acc: 0.600000; val_acc: 0.271000\n",
      "(Epoch 5 / 25) train acc: 0.710000; val_acc: 0.298000\n",
      "(Epoch 6 / 25) train acc: 0.720000; val_acc: 0.282000\n",
      "(Epoch 7 / 25) train acc: 0.832000; val_acc: 0.255000\n",
      "(Epoch 8 / 25) train acc: 0.874000; val_acc: 0.267000\n",
      "(Epoch 9 / 25) train acc: 0.900000; val_acc: 0.277000\n",
      "(Epoch 10 / 25) train acc: 0.900000; val_acc: 0.261000\n",
      "(Epoch 11 / 25) train acc: 0.942000; val_acc: 0.273000\n",
      "(Epoch 12 / 25) train acc: 0.964000; val_acc: 0.305000\n",
      "(Epoch 13 / 25) train acc: 0.980000; val_acc: 0.312000\n",
      "(Epoch 14 / 25) train acc: 0.968000; val_acc: 0.316000\n",
      "(Epoch 15 / 25) train acc: 0.956000; val_acc: 0.293000\n",
      "(Epoch 16 / 25) train acc: 0.988000; val_acc: 0.305000\n",
      "(Epoch 17 / 25) train acc: 0.968000; val_acc: 0.303000\n",
      "(Epoch 18 / 25) train acc: 0.978000; val_acc: 0.303000\n",
      "(Epoch 19 / 25) train acc: 0.986000; val_acc: 0.302000\n",
      "(Epoch 20 / 25) train acc: 0.976000; val_acc: 0.293000\n",
      "(Iteration 101 / 125) loss: 0.043493\n",
      "(Epoch 21 / 25) train acc: 0.988000; val_acc: 0.282000\n",
      "(Epoch 22 / 25) train acc: 0.984000; val_acc: 0.279000\n",
      "(Epoch 23 / 25) train acc: 0.982000; val_acc: 0.294000\n",
      "(Epoch 24 / 25) train acc: 0.994000; val_acc: 0.297000\n",
      "(Epoch 25 / 25) train acc: 0.990000; val_acc: 0.283000\n",
      "\n",
      "0.25\n",
      "(Iteration 1 / 125) loss: 17.318480\n",
      "(Epoch 0 / 25) train acc: 0.230000; val_acc: 0.177000\n",
      "(Epoch 1 / 25) train acc: 0.378000; val_acc: 0.243000\n",
      "(Epoch 2 / 25) train acc: 0.404000; val_acc: 0.254000\n",
      "(Epoch 3 / 25) train acc: 0.496000; val_acc: 0.276000\n",
      "(Epoch 4 / 25) train acc: 0.526000; val_acc: 0.300000\n",
      "(Epoch 5 / 25) train acc: 0.576000; val_acc: 0.301000\n",
      "(Epoch 6 / 25) train acc: 0.618000; val_acc: 0.296000\n",
      "(Epoch 7 / 25) train acc: 0.658000; val_acc: 0.307000\n",
      "(Epoch 8 / 25) train acc: 0.694000; val_acc: 0.316000\n",
      "(Epoch 9 / 25) train acc: 0.728000; val_acc: 0.299000\n",
      "(Epoch 10 / 25) train acc: 0.730000; val_acc: 0.301000\n",
      "(Epoch 11 / 25) train acc: 0.794000; val_acc: 0.320000\n",
      "(Epoch 12 / 25) train acc: 0.780000; val_acc: 0.277000\n",
      "(Epoch 13 / 25) train acc: 0.824000; val_acc: 0.307000\n",
      "(Epoch 14 / 25) train acc: 0.788000; val_acc: 0.321000\n",
      "(Epoch 15 / 25) train acc: 0.832000; val_acc: 0.332000\n",
      "(Epoch 16 / 25) train acc: 0.846000; val_acc: 0.320000\n",
      "(Epoch 17 / 25) train acc: 0.846000; val_acc: 0.283000\n",
      "(Epoch 18 / 25) train acc: 0.892000; val_acc: 0.335000\n",
      "(Epoch 19 / 25) train acc: 0.906000; val_acc: 0.331000\n",
      "(Epoch 20 / 25) train acc: 0.896000; val_acc: 0.322000\n",
      "(Iteration 101 / 125) loss: 3.692541\n",
      "(Epoch 21 / 25) train acc: 0.900000; val_acc: 0.311000\n",
      "(Epoch 22 / 25) train acc: 0.900000; val_acc: 0.319000\n",
      "(Epoch 23 / 25) train acc: 0.898000; val_acc: 0.301000\n",
      "(Epoch 24 / 25) train acc: 0.904000; val_acc: 0.319000\n",
      "(Epoch 25 / 25) train acc: 0.898000; val_acc: 0.301000\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Train two identical nets, one with dropout and one without\n",
    "np.random.seed(231)\n",
    "num_train = 500\n",
    "small_data = {\n",
    "    'X_train': data['X_train'][:num_train],\n",
    "    'y_train': data['y_train'][:num_train],\n",
    "    'X_val': data['X_val'],\n",
    "    'y_val': data['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [1, 0.25]\n",
    "for dropout in dropout_choices:\n",
    "    model = FullyConnectedNet([500], dropout=dropout)\n",
    "    print(dropout)\n",
    "\n",
    "    solver = Solver(model, small_data,\n",
    "                    num_epochs=25, batch_size=100,\n",
    "                    update_rule='adam',\n",
    "                    optim_config={\n",
    "                        'learning_rate': 5e-4,\n",
    "                    },\n",
    "                    verbose=True, print_every=100)\n",
    "    solver.train()\n",
    "    solvers[dropout] = solver\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA4IAAAJNCAYAAABkwMu0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XucXHWZ4P/PY2ikRUy4iZAOECQbxCQQtwERBpAIQSVcvERQQZxRRGEyv9mZjGF3zMbsOLKEEX8szLhxVMALkGFiDA4SFUUHL0jHYEeCkYBIusOM4ZII2kAIz/5R1aHS6U7S6a46XVWf9+vVr6rznG+d81SlKOqp7/d8v5GZSJIkSZKax8uKTkCSJEmSVFsWgpIkSZLUZCwEJUmSJKnJWAhKkiRJUpOxEJQkSZKkJmMhKEmSJElNxkJQktQUImJURDwTEQcXnYskSUUL1xGUJI1EEfFMxeYrgOeAzeXtj2TmV2uflSRJjcFCUJI04kXEI8CHMvO722mzW2a+ULusaqeRn5skqRgODZUk1aWI+LuIuCUiboqIp4H3R8TxEfHTiNgQEY9FxDUR0VJuv1tEZEQcWt7+Snn/tyLi6Yj4SUSMH+BcL4uIWyPiP8rHvisiXlex/xURcXVEPBoRGyPihxHx8vK+k8o5bYyItRFxQTl+d0RcVHGMD0XEXX1y/VhErAF+VY5fGxFdEfH7iLg3It5U8fjdIuITEfFQeX9HRBwUEf83Iv53n+fzrYi4bOj/CpKkemUhKEmqZ+cCXwNGA7cALwB/AewHnACcAXxkO49/L/AJYB/gUeB/baftN4EJwGuAXwJfrth3NTAFOK58rP8OvFguLP8N+AywLzAVWDmI53cWcAwwubx9T/k8+wC3Av/SW3ACs4F3UXrOY4APAc8CNwDvjYiXAUTEAcDJwM2DyEOS1GAsBCVJ9ezuzLwtM1/MzJ7MvDcz78nMFzLzYWAhpaJnILdmZkdmbgK+ChzdX6Py8a/PzKcz81lgHvBfI2LPiBgFXATMyszHMnNzZt5dPub7gTsyc1E5p8cz875BPL+/z8ynMrOnnMeXM/PJ8jDRK4FXAYeX234I+O+Z+WA53/vKbX8M9FS8DucD383MxweRhySpwVgISpLq2drKjYg4IiL+rTyE8/fAfEq9gwP5j4r7fwRe2V+j8oyjV0bEw+Xjrinv2g84ANgdeKifh44bIL6z+j6/v4mIX0XERuApYE9een7bO9eNlIpSyrdfHqCdJKlJWAhKkupZ3xnP/i+lYZuHZ+argLlADMN5LgTeBpxKaRhqby9cAP8JPA+8tp/HrR0gDvAHSrOh9npNP222PL+IeDPw34B3Uhr6uTfwDC89v+2d68vAOyJiarnNbQO0kyQ1CQtBSVIj2QvYCPyhPJnL9q4PHOxxnwOeoFS8fap3R2ZuBq4HPhsRryn3Hp5QnqTmK8AZEfHO8mQu+0XEUeWH3ge8MyJaI+K/AH+6Ezm8ADwOtFAanrpnxf5/Bv4uIl4bJUdHxD7lHH9bPt8NwL+Uh7dKkpqYhaAkqZH8FfAB4GlKvYO3DNNxvwSsK//dD/y4z/6/BB4AlgNPAn9PaYmm3wAzgI+X4z/npYlfrqLU4/c74IuUisbtuR34LvAg8Ajwe+Cxiv0LgCXAneV9C4E9KvbfUD63w0IlSa4jKElSM4iIU4EvAIel//OXpKZnj6AkSQ0uInantKzG5y0CJUlgIShJUkOLiMmUZhjdB7im4HQkSSOEQ0MlSZIkqcnYIyhJkiRJTcZCUJIkSZKazG5FJzBc9ttvvzz00EOLTkOSJEmSCrF8+fLHM3P/nWnbMIXgoYceSkdHR9FpSJIkSVIhIuK3O9vWoaGSJEmS1GQsBCVJkiSpyVgISpIkSVKTKaQQjIgvRsTvIuKXA+yPiLgmItZERGdEvKHWOUqSJElSoyqqR/B64Izt7H8rMKH8dzHwTzXISZIkSZKaQiGFYGb+EHhyO03OBm7Mkp8CYyLiwNpkJ0mSJEmNbaQuHzEWWFux3VWOPVZMOpIkSZJGiiUrulmwbDXrNvRw0JhWZk+fyDlTxxadVl0ZqYVg9BPLbRpFXExp6CgHH3xwtXOSJEmSVLAlK7q5fPFKejZtBqB7Qw+XL14JUPVisJEK0JFaCHYB4yq224B1fRtl5kJgIUB7e/s2haIkSVKvRvoCVw+a8fVuxudchAXLVm8pAnv1bNrMgmWrq/p6F1mAVsNILQSXApdFxM3AccDGzHRYqCRJ2iWN9gVupGvG19teqtpZt6FnUPHhUlQBWi1FLR9xE/ATYGJEdEXEn0XEJRFxSbnJ7cDDwBrg88DHishTkqRqW7KimxOu+B7j5/wbJ1zxPZas6C46pYa0vS9wGn7N+HoX9Zx7C9DuDT0kLxWgtfosKeIz7KAxrYOKD5eiCtBqKaRHMDPP38H+BC6tUTqSJBWiGXtNitJoX+B2VlE9RUW+3s32nIvspSrqM2z29IlbnRegtWUUs6dPrNo5oVRodvfz71ntArRailpHUJKkpteMvSZFKaoHoUhF9hQV9Xo343Musugu6jPsnKlj+fQ7JjN2TCsBjB3TyqffMbnqhe/s6RNpbRm1VawWBWi1WAhKklSQZu2lKkKjfYHbGUX+0FDU692Mz7nIHzmK/Aw7Z+pYfjTnVH5zxdv50ZxTa9LrW1QBWi0jdbIYSZIaXtHDjJppgone51XU8y3itS76SzrU/vVuxudc1DBJKP4zrAjnTB3bMJ+TFoKSJFHMF/Uiv8A14/WJRX2BK+q1LvpLehGvdzM+5yJ/5CjyM0xDZyEoSWp6RX1RL/ILXKNNgz6SFfVaN+OX9GZ8zlDcjxxF97RraCwEJUlNr8iiqKgvcM04q2NRinqtm/FLejM+56I10lDJZmMhKElqes04aUtRQ+iacUhqkcMVm/FLejM+Z2lXOGuoJKnpNePSAs04q2NRmnHGUkkjn4WgJKnpNeMX9aKmQW/G3tdGm3JeUmNwaKgkacQo6tqxZr2uqBlndSyKwxUljTQWgpKkEaHoa8f8ol4bzTqroySNNA4NlSSNCM147VgzcpikJI0M9ghKkkaEZrx2rFnZ+ypJxbNHUJI0IjTjzJ2SJBXFQlCSNCI048ydkiQVxaGhkqQRoVln7pQkqQgWgpKkEcNrxyRJqg2HhkqSJElSk7EQlCRJkqQmYyEoSZKk4de5CK6eBPPGlG47FxWdkaQKhRSCEXFGRKyOiDURMaef/YdExJ0R0RkRd0VEWxF5SpIkaRd0LoLbZsHGtUCWbm+bZTGo4eMPDUNW88liImIUcB1wGtAF3BsRSzNzVUWzq4AbM/OGiDgV+DRwQa1zlaQiLVnRXdgMmkWeW1IDuHM+bOrZOrappxSfMrOYnGqhc1HpOW7sgtFtMG1ubZ5vUect6ty9PzT0vsd6f2iA2py7qNd6mBXRI3gssCYzH87M54GbgbP7tDkSuLN8//v97JekhrZkRTeXL15J94YeEuje0MPli1eyZEV3Q59bUoPY2DW4+HAqqqeoqF7QIntfizr39n5oqKYG6+kuohAcC6yt2O4qxyr9Anhn+f65wF4RsW8NcpOkEWHBstX0bNq8Vaxn02YWLFvd0OeWNMyKKopGD3BVz0Dx4VLkF/WiipOizlvkuYv6oaHI17oKiigEo59Y9tn+a+DkiFgBnAx0Ay9sc6CIiyOiIyI61q9fP/yZSlJB1m3oGVS8Uc4taRgVWRRNmwstrVvHWlpL8Woq8ot6UcVJkb2vRZ27qB8ainytq6CIQrALGFex3Qasq2yQmesy8x2ZORX4H+XYxr4HysyFmdmeme37779/NXOWpJo6aEzroOKNcm5Jw6jIomjKTJhxDYweB0TpdsY11b+Wqsgv6kUVJ0Wdt8hzF/VDQ5GvdRUUUQjeC0yIiPERsTtwHrC0skFE7BcRvbldDnyxxjlKUqFmT59Ia8uorWKtLaOYPX1iQ59b0jAquvdiykz4y1/CvA2l21pMqFHkF/WiipOizlvkuYv6oaHI17oKaj5raGa+EBGXAcuAUcAXM/P+iJgPdGTmUuAU4NMRkcAPgUtrnackFal3hs4iZu4s8tyShtHotvKw0H7ijWra3K1nk4TafVHvLUJqPaNkUecdCeeu9WydRT7fKojMvpfn1af29vbs6OgoOg1JDcilFCTVpb5T7EOpKKpFz0mRGmh6f2mwImJ5ZrbvTNua9whKUj3pXUqhdxbN3qUUAItBSSNbg/Ve7LQieoqkOmQhKEnbsb2lFCwEJY14FkWSBlDEZDGSVDdcSkGSJDUiC0FJ2g6XUpAkSY3IQlCStsOlFCRJUiPyGkFJ2g6XUpAkSY3IQlCSduCcqWMt/CRJUkNxaKgkSWoOnYvg6kkwb0zptnNRc5xbkvphj6AkSWp8fRdX37i2tA3VX16hyHNL0gDsEZQkSY3vzvkvFWK9NvWU4o18bkkagIWgJElqfBu7BhdvlHNL0gAcGiqpbixZ0e3snVIj6FxU6g3b2AWj22Da3OoPkRzdVhqS2V+82oo8tyQNwB5BSXVhyYpuLl+8ku4NPSTQvaGHyxevZMmK7qJTkzQYvdfLbVwL5EvXy1V78pRpc6GldetYS2spXm1FnluSBmAhKKkuLFi2mp5Nm7eK9WzazIJlqwvKSNIuKep6uSkzYcY1MHocEKXbGdfUZrKWIs8tSQNwaKikurBuQ8+g4lLdKGKYZJGKvF5uysziXtsizy1J/bBHUFJdOGhM66DiUl0oaphkkQa6Ls7r5SSppiwEJdWF2dMn0toyaqtYa8soZk+fWFBG0jBoxmUFvF5OkkYEh4ZKqgu9s4M6a6gaSjMuK9A7PLKZhsNK0ghkISipbpwzdayFnxpLsy4r4PVyklS4QoaGRsQZEbE6ItZExJx+9h8cEd+PiBUR0RkRbysiT0mSqsphkpKkgtS8EIyIUcB1wFuBI4HzI+LIPs3+FliUmVOB84B/rG2WkiTVgMsKSJIKUsTQ0GOBNZn5MEBE3AycDayqaJPAq8r3RwPrapqhJEm14jBJSVIBiigExwKVF0R0Acf1aTMP+HZE/DmwJ/CW2qQmSSpUs62pJ0lSQYq4RjD6iWWf7fOB6zOzDXgb8OWI2CbXiLg4IjoiomP9+vVVSFWSVDPNuKaeJEkFKaIQ7ALGVWy3se3Qzz8DFgFk5k+APYD9+h4oMxdmZntmtu+///5VSleSVBPNuKZe0ToXwdWTYN6Y0q1FtyQ1jSIKwXuBCRExPiJ2pzQZzNI+bR4FpgFExOsoFYJ2+UlSI2vGNfWKZA+sJDW1mheCmfkCcBmwDHiA0uyg90fE/Ig4q9zsr4APR8QvgJuAizKz7/BRSVIjGWjtvFqtqddsvWP2wEpSUytkQfnMvB24vU9sbsX9VcAJtc5LklSgaXNLPVKVxUmt1tTr7R3rPXdv7xg07mQ19sBKUlMrZEF5SZK2UeSaes3YO1Z0D6wkqVCF9AhKktSvotbUa8besSJ7YCVJhbNHUJKkZuwdK7IHVpJUOHsEJUlq1t6xonpgJUmFs0dQkiR7xyRJTcYeQUmSwN4xSVJTsUdQkiRJkpqMhaAkSZIkNRkLQUmSJElqMhaCkiRJktRkLAQlSZIkqclYCEqSJElSk7EQlCRtq3MRXD0J5o0p3XYuKjojSZI0jFxHUJJGqs5FcOd82NgFo9tg2tzarHPXuQhumwWbekrbG9eWtsF19iRJahD2CErSSNRbjG1cC+RLxVgteubunP9SEdhrU08pLkmSGoI9gpIGZcmKbhYsW826DT0cNKaV2dMncs7UsUWn1Xi2V4xVu1duY9fg4pIkqe7YIyhppy1Z0c3li1fSvaGHBLo39HD54pUsWdFddGqNp8hibHTb4OKSJKnuWAhK2mkLlq2mZ9PmrWI9mzazYNnqgjJqYEUWY9PmQkvr1rGW1lJckiQ1BAtBqU4tWdHNCVd8j/Fz/o0TrvheTXrl1m3oGVRcQ1BkMTZlJsy4BkaPA6J0O+MaJ4qRJKmBeI2gVId6h2j29s71DtEEqnq93kFjWunup+g7aExrP601JL1FVxGzhvae38JPkqSGVUiPYEScERGrI2JNRMzpZ//VEXFf+e/XEbGhiDylkaqoIZqzp0+ktWXUVrHWllHMnj6xqudtWlNmwl/+EuZtKN1amEmSpGFS8x7BiBgFXAecBnQB90bE0sxc1dsmM/+yov2fA1Nrnac0khU1RLO3t9FZQyVJkupbEUNDjwXWZObDABFxM3A2sGqA9ucD/7NGuUl1ocghmudMHWvhJ0mSVOeKGBo6Flhbsd1Vjm0jIg4BxgPfq0FeUt1wiGaNdS6CqyfBvDGl21os6i5JklRFRfQIRj+xHKDtecCtmbm5v50RcTFwMcDBBx88PNlJdcAhmjXUuQhum/XS4u4b15a2wWv2JElS3SqiEOwCxlVstwHrBmh7HnDpQAfKzIXAQoD29vaBikmpITlEs0bunP9SEdhrU08pbiEoSZLqVBFDQ+8FJkTE+IjYnVKxt7Rvo4iYCOwN/KTG+UnSSzZ2DS4uSZJUB2peCGbmC8BlwDLgAWBRZt4fEfMj4qyKpucDN2emPX2SijO6bXBxSZKkOjCkoaERcRnw1cx8ajCPy8zbgdv7xOb22Z43lNwkaVhMm7v1NYIALa2luCRJUp0aao/gayitA7iovEh8fxPBSFL9mjITZlwDo8cBUbqdcY3XB0qSpLoWQx15WS7+Tgc+CLQDi4AvZOZDQ09v57W3t2dHR0ctTylJkiRJI0ZELM/M9p1pO+RrBMvX8P1H+e8FShO83BoRVw712JIkSZKk4TfUawRnAR8AHgf+GZidmZsi4mXAg8DfDD1FSZIkSdJwGuo6gvsB78jM31YGM/PFiDhziMeWpK11Liqt37exqzRr57S5XqsnSZK0C4Y6NPR24MnejYjYKyKOA8jMB4Z4bEl6Seei0uydG9cCWbq9bVYpLkmSpEEZaiH4T8AzFdt/KMckaXjdOX/rJRygtH3n/GLykSRJqmNDLQSjcsH3zHyRoQ83laRtbewaXFySJEkDGmoh+HBEzIqIlvLfXwAPD0dikrSV0W2Di0uSJGlAQy0ELwHeBHQDXcBxwMVDTUqStjFtLrS0bh1raS3FJUmSNChDGsaZmb8DzhumXCRpYL2zgzprqCRJ0pANdR3BPYA/A14P7NEbz8w/HWJekrStKTMt/CRJkobBUIeGfhl4DTAd+AHQBjw91KQkSZIkSdUz1ELw8Mz8BPCHzLwBeDsweehpSZIkSZKqZaiF4Kby7YaImASMBg4d4jElSZIkSVU01DX/FkbE3sDfAkuBVwKfGHJWkkauzkVO2CJJklTndrkQjIiXAb/PzKeAHwKHDVtWkkamzkVw2yzY1FPa3ri2tA0Wg5IkSXVkl4eGZuaLwGXDmIukke7O+S8Vgb029ZTikiRJqhtDvUbwOxHx1xExLiL26f0blswkjTwbuwYXlyRJ0og01GsEe9cLvLQiljhMVGpMo9tKw0H7i0uSJKluDKlHMDPH9/O3wyIwIs6IiNURsSYi5gzQZmZErIqI+yPia0PJU9IwmTYXWlq3jrW0luKSJEmqG0PqEYyIC/uLZ+aN23nMKOA64DSgC7g3IpZm5qqKNhOAy4ETMvOpiHj1UPKUqmXJim4WLFvNug09HDSmldnTJ3LO1LFFp1U9vRPCOGuoJElSXRvq0NBjKu7vAUwDfg4MWAgCxwJrMvNhgIi4GTgbWFXR5sPAdeUZScnM3w0xT2nYLVnRzeWLV9KzaTMA3Rt6uHzxSoDGLwYt/CRJkurakArBzPzzyu2IGA18eQcPGwtUXmTUBRzXp81/KR/vR8AoYF5m3jGUXKXhtmDZ6i1FYK+eTZtZsGx1YxeCkiRJqntD7RHs64/AhB20iX5i2Wd7t/JxTgHagH+PiEmZuWGrA0VcDFwMcPDBB+9KvtIuW7ehZ1BxSZIkaaQY6jWCt/FSEfcy4Ehg0Q4e1gWMq9huA9b10+anmbkJ+E1ErKZUGN5b2SgzFwILAdrb2/sWk1JVHTSmle5+ir6DxrT201qSJEkaOYbaI3hVxf0XgN9m5o4WFLsXmBAR44Fu4DzgvX3aLAHOB66PiP0oDRV9eIi5SsNq9vSJW10jCNDaMorZ0ycWmJUkSZK0Y0MtBB8FHsvMZwEiojUiDs3MRwZ6QGa+EBGXAcsoXf/3xcy8PyLmAx2ZubS87/SIWAVsBmZn5hNDzFUaVr3XARY2a2jnImfvlCRJ0i6JzF0fURkRHcCbMvP58vbuwI8y85jtP3L4tbe3Z0dHR61PKxWjcxHcNgs2VQxNbWmFGddYDEqSJDWpiFieme0703ZIC8oDu/UWgQDl+7sP8ZiSduTO+VsXgVDavnN+MflIkiSprgy1EFwfEWf1bkTE2cDjQzympB3ZOMCluAPFJUmSpApDvUbwEuCrEXFtebsLuHCIx5S0I6PbYOPa/uOSJEnSDgypRzAzH8rMN1JaNuL1mfmmzFwzPKlJGtC0uaVrAiu1tJbikiRJ0g4MqRCMiL+PiDGZ+UxmPh0Re0fE3w1XcpIGMGVmaWKY0eOAKN06UYwkSZJ20lBnDV2RmVP7xH6emW8YcmaD5KyhkiRJkppZLWcNHRURL684cSvw8u20lyRJkiQVbKiTxXwFuDMivlTe/iBwwxCPKUmSJEmqoiEVgpl5ZUR0Am8BArgDOGQ4EpMkSZIkVcdQh4YC/AfwIvBOYBrwwDAcU5IkSZJUJbvUIxgR/wU4DzgfeAK4hdLEM28extwkSZIkSVWwq0NDfwX8OzCjd93AiPjLYctKGqQlK7pZsGw16zb0cNCYVmZPn8g5U8cWnZYkSZI0Iu3q0NB3UhoS+v2I+HxETKN0jaBUc0tWdHP54pV0b+ghge4NPVy+eCVLVnQXnZokSZI0Iu1SIZiZX8/M9wBHAHcBfwkcEBH/FBGnD2N+0g4tWLaank2bt4r1bNrMgmWrq3/yzkVw9SSYN6Z027mo+ueUJEmShmhIk8Vk5h8y86uZeSbQBtwHzBmWzKSdtG5Dz6Diw6ZzEdw2CzauBbJ0e9ssi0FJkiSNeMMxaygAmflkZv7fzDx1uI4p7YyDxrQOKj5s7pwPm/oUm5t6SnFJkiRpBBu2QlAqyuzpE2ltGbVVrLVlFLOnT6zuiTd2DS4uSZIkjRBDWlBeGgl6Zwet+ayho9vKw0L7iUuSJEkjmIWgGsI5U8fWfrmIaXNL1wRWDg9taS3FJUmSpBGskKGhEXFGRKyOiDURsc3kMhFxUUSsj4j7yn8fKiJPabumzIQZ18DocUCUbmdcU4pLkiRJI1jNewQjYhRwHXAa0AXcGxFLM3NVn6a3ZOZltc5PGpQpMy38JEmSVHeK6BE8FliTmQ9n5vPAzcDZBeShRuJ6fpIkSdJOK6IQHAtUzrDRVY719c6I6IyIWyNiXG1SU11yPT9JkiRpUIooBKOfWPbZvg04NDOnAN8Fbuj3QBEXR0RHRHSsX79+mNNU3XA9P0mSJGlQiigEu4DKHr42YF1lg8x8IjOfK29+Hviv/R0oMxdmZntmtu+///5VSVZ1wPX8JEmSpEEpYvmIe4EJETEe6AbOA95b2SAiDszMx8qbZwEP1DZF7YolK7prv5YfuJ6fJEmSNEg17xHMzBeAy4BllAq8RZl5f0TMj4izys1mRcT9EfELYBZwUa3z1OAsWdHN5YtX0r2hhwS6N/Rw+eKVLFnRXf2TT5tbWr+vkuv5SZIkSQOKzL6X59Wn9vb27OjoKDqNpnXCFd+je0PPNvGxY1r50ZxTq59A56LSNYEbu0o9gdPmuqyDJEmSmkpELM/M9p1pW8TQUDWgdf0UgduLDzvX85MkSZJ2WhGTxagBHTSmdVBxSZIkScWxENSwmD19Iq0to7aKtbaMYvb0iQVlJEmSJGkgDg3VsOidHbSQWUMlSZIkDYqFoIbNOVPHWvhJkiRJdcChoZIkSZLUZCwEJUmSJKnJWAhKkiRJUpOxEJQkSZKkJmMhKEmSJElNxkJQkiRJkpqMhaAkSZIkNRkLQUmSJElqMhaCkiRJktRkLAQlSZIkqclYCEqSJElSk9mt6AQ0/Jas6GbBstWs29DDQWNamT19IudMHVt0WpIkSZJGCAvBBrNkRTeXL15Jz6bNAHRv6OHyxSsBLAYlSZIkAQ4NbTgLlq3eUgT26tm0mQXLVheUkSRJkqSRxkKwwazb0DOo+LDqXARXT4J5Y0q3nYuqf05JkiRJg1ZIIRgRZ0TE6ohYExFzttPuXRGREdFey/zq2UFjWgcVHzadi+C2WbBxLZCl29tmWQxKkiRJI1DNC8GIGAVcB7wVOBI4PyKO7KfdXsAs4J7aZljfZk+fSGvLqK1irS2jmD19YnVPfOd82NSn13FTTykuSZIkaUQpokfwWGBNZj6cmc8DNwNn99PufwFXAs/WMrl6d87UsXz6HZMZO6aVAMaOaeXT75hc/YliNnYNLi5JkiSpMEXMGjoWWFux3QUcV9kgIqYC4zLzmxHx17VMrhGcM3Vs7WcIHd1WHhbaT1ySJEnSiFJEj2D0E8stOyNeBlwN/NUODxRxcUR0RETH+vXrhzFFDdq0udDS5zrEltZSXJIkSdKIUkQh2AWMq9huA9ZVbO8FTALuiohHgDcCS/ubMCYzF2Zme2a277///lVMWTs0ZSbMuAZGjwOidDvjmlJckiRJ0ohSxNDQe4EJETEe6AbOA97buzMzNwL79W5HxF3AX2dmR43z1GBNmWnhJ0mSJNWBmvcIZuYLwGXAMuABYFFm3h8R8yPirFrn05Bcz0+SJEnSdhTRI0hm3g7c3ifW78VkmXlKLXJqGL3r+fUu5dC7nh/YWydJkiQJKGhBeVWR6/lJkiRJ2gELwUbjen6SJEmSdsBCsNEMtG6f6/lJkiRJKrMQbDSu5ydJkiRpBywEG43r+UmSJEnagUJmDVWVuZ6fJEmSpO2wR1CSJEmSmoyFoCRJkiQ1GQtBSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJkiRJTcZCUJIkSZKajIWgJEmSJDUZC0FJkiRJajIWgpIkSZLUZCwEJUnxA47QAAAgAElEQVSSJKnJFFIIRsQZEbE6ItZExJx+9l8SESsj4r6IuDsijiwiT0mSJElqRDUvBCNiFHAd8FbgSOD8fgq9r2Xm5Mw8GrgS+EyN05QkSZKkhlVEj+CxwJrMfDgznwduBs6ubJCZv6/Y3BPIGuYnSZIkSQ1ttwLOORZYW7HdBRzXt1FEXAr8N2B34NTapCZJkiRJja+IHsHoJ7ZNj19mXpeZrwU+DvxtvweKuDgiOiKiY/369cOcpiRJkiQ1piIKwS5gXMV2G7BuO+1vBs7pb0dmLszM9sxs33///YcxRUmSJElqXEUUgvcCEyJifETsDpwHLK1sEBETKjbfDjxYw/wkSZIkqaHV/BrBzHwhIi4DlgGjgC9m5v0RMR/oyMylwGUR8RZgE/AU8IFa5ylJkiRJjaqIyWLIzNuB2/vE5lbc/4uaJyVJkiRJTaKQBeUlSZIkScWxEJQkSZKkJmMhKEmSJElNppBrBCVJktR8Nm3aRFdXF88++2zRqUh1bY899qCtrY2WlpZdPoaFYJUsWdHNgmWrWbehh4PGtDJ7+kTOmTq26LQkSZIK09XVxV577cWhhx5KRBSdjlSXMpMnnniCrq4uxo8fv8vHcWhoFSxZ0c3li1fSvaGHBLo39HD54pUsWdFddGqSJEmFefbZZ9l3330tAqUhiAj23XffIfesWwhWwYJlq+nZtHmrWM+mzSxYtrqgjCRJkkYGi0Bp6IbjvyMLwSpYt6FnUHFJkiTVxp/+6Z/y6le/mkmTJg3YJjOZNWsWhx9+OFOmTOHnP//5ln033HADEyZMYMKECdxwww07PN9dd93FmWeeOSy574rrr7+edevWFXb+4XbHHXcwceJEDj/8cK644op+23zmM5/hyCOPZMqUKUybNo3f/va3W/aNGjWKo48+mqOPPpqzzjprh+d75JFHtvteqbYlS5awatWqqhzbQrAKDhrTOqi4JEmSauOiiy7ijjvu2G6bb33rWzz44IM8+OCDLFy4kI9+9KMAPPnkk3zyk5/knnvu4Wc/+xmf/OQneeqpp3YpjxdeeGGXHjdYjVQIbt68mUsvvZRvfetbrFq1iptuuqnfImnq1Kl0dHTQ2dnJu971Lv7mb/5my77W1lbuu+8+7rvvPpYuXbrLudTq389CsM7Mnj6R1pZRW8VaW0Yxe/rEgjKSJEmqP0tWdHPCFd9j/Jx/44Qrvjcs8y2cdNJJ7LPPPttt841vfIMLL7yQiOCNb3wjGzZs4LHHHmPZsmWcdtpp7LPPPuy9996cdtpp/RaVd9xxB0cccQQnnngiixcv3hKfN28eF198MaeffjoXXnghzz77LB/84AeZPHkyU6dO5fvf/z5QKt7OPvtszjjjDCZOnMgnP/nJLcf4zGc+w6RJk5g0aRKf/exngW17ra666irmzZvHrbfeSkdHB+973/s4+uij6emp8ei0zkVw9SSYN6Z027loSIf72c9+xuGHH85hhx3G7rvvznnnncc3vvGNbdq9+c1v5hWveAUAb3zjG+nq6hrUeZYvX85RRx3F8ccfz3XXXbclfv311/Pud7+bGTNmcPrpp5OZzJ49m0mTJjF58mRuueUWoNQLfNJJJ3Huuedy5JFHcskll/Diiy8CcNNNNzF58mQmTZrExz/+8S3HfuUrX7nl/q233spFF13Ej3/8Y5YuXcrs2bM5+uijeeihhwb1PHbEWUOroHd2UGcNlSRJ2jW9k+/1zrvQO/keUPXvVN3d3YwbN27LdltbG93d3QPGKz377LN8+MMf5nvf+x6HH34473nPe7bav3z5cu6++25aW1v5h3/4BwBWrlzJr371K04//XR+/etfA6Wi55e//CWveMUrOOaYY3j7299ORPClL32Je+65h8zkuOOO4+STT2bvvffu93m8613v4tprr+Wqq66ivb19WF6bnda5CG6bBZvKxefGtaVtgCkzd+mQ/b3+99xzz3Yf84UvfIG3vvWtW7afffZZ2tvb2W233ZgzZw7nnHPONo/54Ac/yP/5P/+Hk08+mdmzZ2+17yc/+QmdnZ3ss88+/Ou//iv33Xcfv/jFL3j88cc55phjOOmkk4DSv9+qVas45JBDOOOMM1i8eDFvetOb+PjHP87y5cvZe++9Of3001myZEm/OQC86U1v4qyzzuLMM8/kXe96106/TjvLQrBKzpk61sJPkiRpF21v8r1qf8fKzG1iETFgvNKvfvUrxo8fz4QJEwB4//vfz8KFC7fsP+uss2htLV0udPfdd/Pnf/7nABxxxBEccsghWwrB0047jX333ReAd7zjHdx9991EBOeeey577rnnlvi///u/79S1bjV35/yXisBem3pK8V0sBHfm9a/0la98hY6ODn7wgx9siT366KMcdNBBPPzww5x66qlMnjyZ1772tVv2b9y4kQ0bNnDyyScDcMEFF/Ctb31ry/7eHmEo/fudf/75jBo1igMOOICTTz6Ze++9l1e96lUce+yxHHbYYQCcf/753H333bS0tHDKKaew//77A/C+972PH/7whwMWgtXm0NBqGeaucEmSpGZS5OR7bW1trF27dst2V1cXBx100IDxvrZXnPQWcdB/YTPQMQYqRAF22223LUMPgSEvKzAsNg4wHHOg+E7Y2dcf4Lvf/S6f+tSnWLp0KS9/+cu3xHvbH3bYYZxyyimsWLFiq8dlZs3//fq2r9W/n4VgNfR2hW9cC+RLXeEWg5IkSTulyMn3zjrrLG688UYyk5/+9KeMHj2aAw88kOnTp/Ptb3+bp556iqeeeopvf/vbTJ8+favHHnHEEfzmN7/Zcj3XTTfdNOB5TjrpJL761a8C8Otf/5pHH32UiRNLc0p85zvf4cknn6Snp4clS5ZwwgkncNJJJ7FkyRL++Mc/8oc//IGvf/3r/Mmf/AkHHHAAv/vd73jiiSd47rnn+OY3v7nlHHvttRdPP/30cL9EOza6bXDxnXDMMcfw4IMP8pvf/Ibnn3+em2++ud/e0BUrVvCRj3yEpUuX8upXv3pL/KmnnuK5554D4PHHH+dHP/oRRx555FaPHTNmDKNHj+buu+8G2PLv05+TTjqJW265hc2bN7N+/Xp++MMfcuyxxwKloaG/+c1vePHFF7nllls48cQTOe644/jBD37A448/zubNm7npppu29DwecMABPPDAA7z44ot8/etf33KOav77WQhWw/a6wiVJkrRD1Zp87/zzz+f4449n9erVtLW18YUvfAGAz33uc3zuc58D4G1vexuHHXYYhx9+OB/+8If5x3/8RwD22WcfPvGJT3DMMcdwzDHHMHfu3G0mntljjz1YuHAhb3/72znxxBM55JBDBszlYx/7GJs3b2by5Mm85z3v4frrr9/Se3XiiSdywQUXcPTRR/POd76T9vZ23vCGN3DRRRdx7LHHctxxx/GhD32IqVOn0tLSwty5cznuuOM488wzOeKII7ac46KLLuKSSy6p/WQx0+ZCS5+ivaW1FN9Fu+22G9deey3Tp0/nda97HTNnzuT1r389AHPnzt0yC+js2bN55plnePe7373VMhEPPPAA7e3tHHXUUbz5zW9mzpw52xSCAF/60pe49NJLOf7447cM4+3Pueeey5QpUzjqqKM49dRTufLKK3nNa14DwPHHH8+cOXOYNGkS48eP59xzz+XAAw/k05/+NG9+85s56qijeMMb3sDZZ58NwBVXXMGZZ57JqaeeyoEHHrjlHOeddx4LFixg6tSpwz5ZTGyvi7KetLe3Z0dHR9FplMwbA/T3ugbM21DrbCRJkkaEBx54gNe97nU73X7Jiu6mnHzv+uuvp6Ojg2uvvbboVIamc1GpI2RjV6kncNrcXb4+sJ7cddddXHXVVVv1zFZDf/89RcTyzNypmYGcLKYaRreVh4X2E5ckSdJOcfK9OjdlZlMUfvXKoaHVUIWucEmSJDWHiy66qP57A5vYKaecUvXewOFgIVgNU2bCjGtg9DggSrczrvEXEUmSJEkjgkNDq8WucEmSpG3saHp+STs2HPO8FNIjGBFnRMTqiFgTEXP62f/fImJVRHRGxJ0RMfB0S5IkSaoLe+yxB0888cSwfImVmlVm8sQTT7DHHnsM6Tg17xGMiFHAdcBpQBdwb0QszcxVFc1WAO2Z+ceI+ChwJfCeWucqSZKk4dPW1kZXVxfr168vOhWpru2xxx60tQ1tIsoihoYeC6zJzIcBIuJm4GxgSyGYmd+vaP9T4P01zVCSJEnDrqWlhfHjxxedhiSKGRo6FqhcW6GrHBvInwHfqmpGkiRJktREiugR7O/q4H4HikfE+4F24OQB9l8MXAxw8MEHD1d+kiRJktTQiugR7ALGVWy3Aev6NoqItwD/AzgrM5/r70CZuTAz2zOzff/9969KspIkSZLUaKLWszZFxG7Ar4FpQDdwL/DezLy/os1U4FbgjMx8cCePux747fBnPGT7AY8XnYQamu8xVZPvL1WT7y9Vk+8vVdNIfX8dkpk71UNW80IQICLeBnwWGAV8MTM/FRHzgY7MXBoR3wUmA4+VH/JoZp5V80SHQUR0ZGZ70XmocfkeUzX5/lI1+f5SNfn+UjU1wvurkAXlM/N24PY+sbkV999S86QkSZIkqUkUsqC8JEmSJKk4FoLVt7DoBNTwfI+pmnx/qZp8f6mafH+pmur+/VXINYKSJEmSpOLYIyhJkiRJTcZCsIoi4oyIWB0RayJiTtH5qLFExCMRsTIi7ouIjqLzUf2LiC9GxO8i4pcVsX0i4jsR8WD5du8ic1T9GuD9NS8iusufY/eVZxWXBi0ixkXE9yPigYi4PyL+ohz3M0xDtp33V11/hjk0tEoiYhSl9RJPA7oorZd4fmauKjQxNYyIeARoz8yRuIaN6lBEnAQ8A9yYmZPKsSuBJzPzivIPWntn5seLzFP1aYD31zzgmcy8qsjcVP8i4kDgwMz8eUTsBSwHzgEuws8wDdF23l8zqePPMHsEq+dYYE1mPpyZzwM3A2cXnJMkDSgzfwg82Sd8NnBD+f4NlP7HJw3aAO8vaVhk5mOZ+fPy/aeBB4Cx+BmmYbCd91ddsxCsnrHA2ortLhrgDaMRJYFvR8TyiLi46GTUsA7IzMeg9D9C4NUF56PGc1lEdJaHjjpsT0MWEYcCU4F78DNMw6zP+wvq+DPMQrB6op+Y43A1nE7IzDcAbwUuLQ+7kqR68k/Aa4GjgceAfyg2HdW7iHgl8K/A/5eZvy86HzWWft5fdf0ZZiFYPV3AuIrtNmBdQbmoAWXmuvLt74CvUxqOLA23/yxfG9F7jcTvCs5HDSQz/zMzN2fmi8Dn8XNMQxARLZS+pH81MxeXw36GaVj09/6q988wC8HquReYEBHjI2J34DxgacE5qUFExJ7li5WJiD2B04Ffbv9R0i5ZCnygfP8DwDcKzEUNpvcLetm5+DmmXRQRAXwBeCAzP1Oxy88wDdlA7696/wxz1tAqKk8h+1lgFPDFzPxUwSmpQUTEYZR6AQF2A77m+0tDFRE3AacA+wH/CfxPYAmwCDgYeBR4d2Y64YcGbYD31ymUhlQl8Ajwkd7ruaTBiIgTgX8HVgIvlsP/ndJ1XH6GaUi28/46nzr+DLMQlCRJkqQm49BQSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJEmSpCZjIShJUh8RsTki7qv4mzOMxz40IupqrSlJUuPZregEJEkagXoy8+iik5AkqVrsEZQkaSdFxCMR8b8j4mflv8PL8UMi4s6I6CzfHlyOHxARX4+IX5T/3lQ+1KiI+HxE3B8R346I1sKelCSpKVkISpK0rdY+Q0PfU7Hv95l5LHAt8Nly7FrgxsycAnwVuKYcvwb4QWYeBbwBuL8cnwBcl5mvBzYA76zy85EkaSuRmUXnIEnSiBIRz2TmK/uJPwKcmpkPR0QL8B+ZuW9EPA4cmJmbyvHHMnO/iFgPtGXmcxXHOBT4TmZOKG9/HGjJzL+r/jOTJKnEHkFJkgYnB7g/UJv+PFdxfzNesy9JqjELQUmSBuc9Fbc/Kd//MXBe+f77gLvL9+8EPgoQEaMi4lW1SlKSpO3xF0hJkrbVGhH3VWzfkZm9S0i8PCLuofRj6vnl2CzgixExG1gPfLAc/wtgYUT8GaWev48Cj1U9e0mSdsBrBCVJ2knlawTbM/PxonORJGkoHBoqSZIkSU3GHkFJkiRJajL2CEqSmkpEHBoRGRFeJy9JaloWgpKkuhIRyyJifj/xsyPiPyzwJEnaMQtBSVK9uR64ICKiT/wC4KuZ+ULtUxoeUeL/myVJVef/bCRJ9WYJsA/wJ72BiNgbOBO4sbz99ohYERG/j4i1ETFvZw8eEXMi4qGIeDoiVkXEuX32fzgiHqjY/4ZyfFxELI6I9RHxRERcW47Pi4ivVDx+q6GpEXFXRHwqIn4E/BE4LCI+WHGOhyPiI31yODsi7is/v4ci4oyIeHdELO/T7q8iYsnOPndJUvOwEJQk1ZXM7AEWARdWhGcCv8rMX5S3/1DePwZ4O/DRiDhnJ0/xEKUiczTwSeArEXEgQES8G5hXPvargLOAJyJiFPBN4LfAocBY4OZBPK0LgIuBvcrH+B2lwvZVlNYkvLqi4DyWUsE7u/z8TgIeAZYC4yPidRXHfT/w5UHkIUlqEhaCkqR6dAPw7ohoLW9fWI4BkJl3ZebKzHwxMzuBm4CTd+bAmfkvmbmu/NhbgAeBY8u7PwRcmZn3ZsmazPxtef9BwOzM/ENmPpuZdw/i+Vyfmfdn5guZuSkz/y0zHyqf4wfAt3mpB/TPgC9m5nfKOXZn5q8y8zngFkrFHxHxekpF6TcHkYckqUlYCEqS6k65yFoPnB0RhwHHAF/r3R8Rx0XE98vDNDcClwD77cyxI+LC8rDLDRGxAZhU8dhxlHoM+xoH/HYI1yeu7ZPDWyPipxHxZDmHt+1EDlAqht9bvn7yAmBRuUCUJGkrFoKSpHp1I6WewAuAb2fmf1bs+xqloZLjMnM08Dmg7+Qy24iIQ4DPA5cB+2bmGOCXFY9dC7y2n4euBQ4eYMbSPwCvqNh+TT9ttizqGxEvB/4VuAo4oJzD7TuRA5n5U+B5Sr2H78VhoZKkAVgISpLq1Y3AW4APUzEstGwv4MnMfLZ8Td17d/KYe1IqytYDRMQHKfUI9vpn4K8j4r+WZ/g8vFw8/gx4DLgiIvaMiD0i4oTyY+4DToqIgyNiNHD5DnLYHXh5OYcXIuKtwOkV+78AfDAipkXEyyJibEQcUbH/RuBa4IVBDk+VJDURC0FJUl3KzEeAH1Mq3pb22f0xYH5EPA3MpTS5zM4ccxXwD8BPgP8EJgM/qtj/L8CnKPU4Pk15BtPM3AzMAA4HHgW6gPeUH/MdStfudQLL2cE1e5n5NDCrnPNTlIrYpRX7f0Z5AhlgI/AD4JCKQ3yZUvFqb6AkaUCRmTtuJUmS6kJ5Ap3fAW/IzAeLzkeSNDLZIyhJUmP5KHCvRaAkaXv6u6hdkiTVoYh4hNKkMju7ZqIkqUk5NFSSJEmSmoxDQyVJkiSpyVgISpIkSVKTqeo1ghFxBvD/A6OAf87MK/rsvwS4FNgMPANcXJ66u3f/wcAqYF5mXrW9c+2333556KGHDu8TkCRJkqQ6sXz58sczc/+daVu1awQjYhTwa+A0Susp3Quc36fQe1Vm/r58/yzgY5l5RsX+fwVeBO7ZUSHY3t6eHR0dw/9EJEmSJKkORMTyzGzfmbbVHBp6LLAmMx/OzOeBm4GzKxv0FoFlewJbqtKIOAd4GLi/ijlKkiRJUtOpZiE4Flhbsd1Vjm0lIi6NiIeAK4FZ5diewMeBT1YxP0mSJElqStUsBKOf2DbjUDPzusx8LaXC72/L4U8CV2fmM9s9QcTFEdERER3r168fcsKSJEmS1AyqOVlMFzCuYrsNWLed9jcD/1S+fxzwroi4EhgDvBgRz2bmtZUPyMyFwEIoXSM4XIlLkiRJUiOrZiF4LzAhIsYD3cB5wHsrG0TEhMx8sLz5duBBgMz8k4o284Bn+haBkiRJkqRdU7VCMDNfiIjLgGWUlo/4YmbeHxHzgY7MXApcFhFvATYBTwEfqFY+kiRJkqSSqi0fUWsuHyFJktRH5yK4cz5s7ILRbTBtLkyZWXRWkqpkMMtHVHVBeUmSJBWkcxHcNgs29ZS2N64tbYPFoKSqzhoqSZKkotw5/6UisNemnlJcUtOzEJQkSWpEG7sGF5fUVCwEJUlSbXUugqsnwbwxpdvORUVn1JhGtw0uLqmpWAhKkqTa6b1ubeNaIF+6bs1icPhNmwstrVvHWlpL8Vqw4JdGNAtBSZJUO163VjtTZsKMa2D0OCBKtzOuqc1EMRb80ojnrKGSJKl2vG6ttqbMLGaG0O0V/M5YKo0I9ghKkqTa8bq15mDBL414FoKSJKl2ir5uTbVhwS+NeBaCkiSpdoq8bk21Y8EvjXheIyhJkmqrqOvWVDu9/753zi8NBx3dVioC/XeXRgwLQUmSJA0/C35pRHNoqCRJkiQ1GQtBSZIkSWoyFoKSJEmS1GQsBCVJkiSpyVgISpIkSVKTsRCUJElS4+hcBFdPgnljSredi4rOSBqRXD5CkqRm1bnIdd7UWDoXwW2zYFNPaXvj2tI2+N6W+qhqj2BEnBERqyNiTUTM6Wf/JRGxMiLui4i7I+LIcvy0iFhe3rc8Ik6tZp6SJDWd3i/MG9cC+dIXZntPVM/unP9SEdhrU08pLmkrVSsEI2IUcB3wVuBI4PzeQq/C1zJzcmYeDVwJfKYcfxyYkZmTgQ8AX65WnpKkEcQhXbXTjF+YfX81vo1dg4tLTayaPYLHAmsy8+HMfB64GTi7skFm/r5ic08gy/EVmbmuHL8f2CMiXl7FXCVJRbOHqraa7Quz76/mMLptcPFG0Yw/cjTjcx5m1SwExwJrK7a7yrGtRMSlEfEQpR7BWf0c553Aisx8ripZSpJGhmbsoSpSs31h9v3VHKbNhZbWrWMtraV4o2rGHzma8TlXQTULwegnltsEMq/LzNcCHwf+dqsDRLwe+N/AR/o9QcTFEdERER3r168fhpQlSYVpth6qojXbF2bfX81hykyYcQ2MHgdE6XbGNY09UUwz/sjRjM+5Cqo5a2gXMK5iuw1YN0BbKA0d/afejYhoA74OXJj5/9q7+zir6zrh/693iDJqC2pkyojhwmIECP0GzDDwJkVT8SbzZiujO7vRdbt6/Fjx18Yq13bJpdb2c3W32KvStlLJVaItJVctFzeNIQgTIk1NZ7S8xc1ERXhff5wzOIwDnGHmO2fOOa/n4zGPc76f8/me8z5zPnzPvPnc5W+7OyEzFwILAVpaWl6XZEqSasjQ5vL/7nZTrr7X8Ydxo6waavtqHBPPqN923J1G/E+ORnzPBSiyR3A5MCYiRkXErsBZwJLOFSJiTKfDE4AHyuXDgB8CF2Xm3QXGKEk75jyE/tFoPVQDwOJN05j28pWMeuk7THv5ShZvmlbtkIpj+1K9arRh3tCY77kAhSWCmfkqcD6wFFgLLMrM+yNifkTMKlc7PyLuj4hVwOcorRBK+bzRwBfKW0usiog3FxWrJG2T8xD6TyMO6aqixSvbueim+2hfv4EE2tdv4KKb7mPxyvZqh1YM25fqVSP+J0cjvucCRGZ9jKhsaWnJ1tbWaochqd78w/htDCc7AP7Hr/o/HqmPTFtwB+3rN7yufMSwJu6e6/a9Uk1Zvahxhnl3aMT3XIGIWJGZLZXULXKOoCTVPuchqE493k0SuL1ySQNYo82LhMZ8z32syDmCklT7nIegOrX/sKYelUuS6ouJoCRtj/MQVKfmzBxL0+BBW5U1DR7EnJljqxSRJKk/OTRUkran0ZbYV79bvLKdy5eu4/H1G9h/WBNzZo7llMkjCn/djteoxmtLkqrPxWIkSaqSjpU7N2zctKWsafAgLj1tggmZVItcwERV1pPFYhwaKklSlVy+dN1WSSDAho2buHzpuipFJGmnud2QaoyJoCRJVeLKnVIduX0+bOzyb3fjhlK5NACZCEqSVCWu3CnVEbcbUo0xEZQkqUpcuVOqI243pBpjIihJUpWcMnkEl542gRHDmghgxLAmF4qRapXbDanGuH2EpNrhamyqQ6cMuptTdpsPQ9pgt2YYNA+wXUs1x+2GVGNMBCXVho7V2Dom4nesxgZ+yap22a6l+jLxDP/tqmaYCEqqDdtbjc0vXfWBqmzsbrvuV1X5jCVpgDIRlFQbXI1NBeq6sXv7+g1cdNN9AMUmCrbrflO1z1iSBigXi5FUG1yNTQWq2sbutut+U7XPWJIGKBNBSbXB1dhUoKpt7G677jdV+4zLFq9sZ9qCOxg194dMW3AHi1e21/XrShr4HBoqqTa4GpsKtP+wJtq7SQgK39jddt1vqvYZU71hqdUeDuucTGlgi8ysdgx9oqWlJVtbW6sdhiSpBnX9gxlKG7u7p1/9qOZnPG3BHd0moSOGNXH33KPq7nXBf1NStUTEisxsqaRuoUNDI+K4iFgXEQ9GxNxuHv9URNwXEasiYllEjOv02EXl89ZFxMwi45Rq0upF8A/j4eJhpdvVi6odkepJg7UvN3avf9X8jKs1LLWaw2Gdk9lAGuz7op7eb2FDQyNiEHA1cAzQBiyPiCWZuaZTte9m5lfL9WcBXwaOKyeEZwFvB/YH/iMi/iIzt76iSNVWrQ3O3XtMRWrQ9nXK5BEmfnWuWp9xtYalVnM4bLXnZKqfNNr3RZ293yJ7BKcCD2bmQ5n5CnA9cHLnCpn5350O9wA6xqmeDFyfmS9n5sPAg+XnkwaOjovB848B+drFoD/+Z2h7e49JvWX7kvrUnJljaRo8aKuypsGDmDNzbF2+Lmw72eyPJFT9qNG+L+rs/RaZCI4AHut03FYu20pEnBcRvwUuAy7oyblSVVXzYuDeY42hWsNPbFC0xK0AACAASURBVF9Sn6rWsNRqDoetZhKqftRo3xd19n6LXDU0uil73co0mXk1cHVE/CXwt8CHKz03Is4FzgUYOXJkr4KVeqyaF4OhzeWeyG7KVR+qOfzE9iX1uWoNS63m6wKuGlrvGu37os7eb5E9gm3AAZ2Om4HHt1P/euCUnpybmQszsyUzW4YPH97LcKUequJG0Mv//K/YkLtuVbYhd2X5n/9V4a+tflLNHmf3tpPUB06ZPIK75x7FwwtO4O65R5kE1qMqfl9UZY/MOvt+LDIRXA6MiYhREbErpcVflnSuEBFjOh2eADxQvr8EOCsidouIUcAY4OcFxir1XBUvBp9dM4YLN36cts1vYnMGbZvfxIUbP85n14zZ8cmqDdXscZ54Bpx0JQw9AIjS7UlX1uREeElSgar0fdGxPUn7+g0kr+2RWXgyWGffj4UNDc3MVyPifGApMAj4RmbeHxHzgdbMXAKcHxHvATYCz1EaFkq53iJgDfAqcJ4rhmrAqeJG0I+v30A7h7PklcO3Ko/+WI2tWiulNppqDz+ZeIafqyRpx6rwfbG97UkK73muo+/HIucIkpk/An7UpWxep/t/vZ1zvwh8sbjopD5QpYtB1ZYEr7Nlkwe0o+dt/buGmh5+IklSX3F7kr5R6IbykopRtdXY6mzZ5AGtzoafSJLqUzXm6rk9Sd8otEdQUjGqthpbnS2bPODV0fATSVL96Zir1zFMs2OuHlDo3yRzZo7d6nXB7Ul2homgVKOqsiR4teetSaoLi1e2u62ACmP76j/Vmqvn9iR9w0RQUuWctyapl6rVg6DGYPvqX9Wcq1etPTLriXMEJVXOeWuSeml7PQhSb9m++pdz9WqbPYKSesZ5a5J6wdX+VCTbV/9yrl5ts0dQkiT1G3sQVCTbV/86ZfIILj1tAiOGNRHAiGFNXHraBIds1gh7BCVJUr+xB0FFsn31P+fq1S4TQUmS1G9c7U9Fsn1JlYvMrHYMfaKlpSVbW1urHYYkSZIkVUVErMjMlkrqOkdQkiRJkhqMiaAkSZIkNRgTQUmSJElqMCaCkiRJktRgTAQlSZIkqcGYCEqSJElSg3EfQUmSpKKtXgS3z4fn22BoMxw9DyaeUe2oJPXQ4pXtdbNPpYmgpB6ppwugJPWL1YvgBxfAxg2l4+cfKx2DyWCd8Tuyvi1e2c5FN93Hho2bAGhfv4GLbroPoCY/Z4eGSqpYxwWwff0GktcugItXtlc7NEkauG6f/1oS2GHjhlK56obfkfXv8qXrtiSBHTZs3MTlS9dVKaLeKTQRjIjjImJdRDwYEXO7efxzEbEmIlZHxO0RcWCnxy6LiPsjYm1EXBkRUWSsknas3i6AktQvnm/rWblqkt+R9e/x9Rt6VD7QFZYIRsQg4GrgeGAccHZEjOtSbSXQkpkTgRuBy8rnvguYBkwExgNTgBlFxSqpMvV2AZSkfjG0uWflqkl+R9a//Yc19ah8oCuyR3Aq8GBmPpSZrwDXAyd3rpCZd2bmi+XDe4COK2ICQ4Bdgd2AwcAfCoxVUgXq7QIoSf3i6HkwuMt1cnBTqVx1o5rfkYtXtjNtwR2MmvtDpi24w+GoBZkzcyxNgwdtVdY0eBBzZo6tUkS9U2QiOAJ4rNNxW7lsWz4G3AKQmT8D7gSeKP8szcy1BcUpqUL1dgGUpH4x8Qw46UoYegAQpduTrnShmDpTre9I5yb2n1Mmj+DS0yYwYlgTAYwY1sSlp02oyYVioNhVQ7ub05fdVoz4INBCefhnRIwG3sZrPYS3RcT0zLyry3nnAucCjBw5so/ClrQtHRc6V0STpB6aeIaJX52r1nfk9uYm+v3c906ZPKJufq9FJoJtwAGdjpuBx7tWioj3AJ8HZmTmy+XiU4F7MvOFcp1bgHcCWyWCmbkQWAjQ0tLSbZKpxuByzf2nni6AA53tup+5z5ukXqrGd6RzE7WzikwElwNjImIU0A6cBfxl5woRMRn4GnBcZj7Z6aFHgU9ExKWUehZnAF8pMFb1gWr90bp4ZTvLbv4nbuB69t/taR5/8U185eazgM/4R7NqVr3tVTTguc+bpBq1/7Am2rtJ+py/rx0pbI5gZr4KnA8sBdYCizLz/oiYHxGzytUuB/YEvhcRqyJiSbn8RuC3wH3AL4FfZuYPiopVvVfN8emrfriQ+bGQ5jc8zRsCmt/wNPNjIat+uLDw15aK4jLk/cx93iTVKOfva2cV2SNIZv4I+FGXsnmd7r9nG+dtAj5ZZGzqW9Ucn/7xV77N7m94Zauy3eMVPv7Kt4FLCn1tqSgO9eln7vMmqUY5f187q9BEUI2jmn+07v+GZ3pULtUCh/r0s6HNpeGg3ZVL0gDn/H3tjCK3j1ADqebeOS81vaVH5VItaNShPlXbC8t93iRJDcZEUH2imn+07n78fF4dNGSrslcHDWH3453bo9pVb3sVVaKqe2G5z5skqcFEZn3sutDS0pKtra3VDqOhVXWpe5d9l2retAV3dDscdsSwJu6ee1QVIpIkqbZExIrMbKmkrnME1WeqOj69Shv1us+b1HdcIEeSpP7j0FBpJ1V1GJtUh6o511iSpEZjIijtJPd5k/pWoy6QI0lSNewwEYyI8yNir/4IRqolDmOT+lYjLpAjSVK1VDJH8C3A8oj4BfANYGnWywozUi+4z5vU99wLS5Kk/rHDHsHM/FtgDPB1YDbwQET8r4j484JjkwY0h7FJkiSpVlW0amhmZkT8Hvg98CqwF3BjRNyWmX9TZIDSQNXRa+GqofXP1WElSVK92eE+ghFxAfBh4Gng/wCLM3NjRLwBeCAzB0TPoPsISipCx+qwnRcGaho8yLlrkiRpwOnJPoKVrBr6JuC0zJyZmd/LzI0AmbkZOLEXcUrSgOfqsJIkqR5Vkgj+CHi24yAi3hgRhwJk5tqiApOkgcDVYSVJUj2qJBH8Z+CFTsd/KpdJUt1zk3NJklSPKkkEo/N2EeUhoRUtMiNJtc7VYSVJUj2qJBF8KCIuiIjB5Z+/Bh4qOjBJGgjc5FySJNWjSnr2PgVcCfwtkMDtwLlFBiVJA4mbnEuSpHqzw0QwM58EztqZJ4+I44D/HxgE/J/MXNDl8c8BH6e0N+FTwEcz83flx0ZS2q7iAEoJ6Hsz85GdiUOS1DPunShJUn3bYSIYEUOAjwFvB4Z0lGfmR3dw3iDgauAYoA1YHhFLMnNNp2orgZbMfDEiPg1cBpxZfuxbwBcz87aI2BPYXPnbkiTtrK57J7av38BFN90HYDIoSVKdqGSO4L8CbwFmAj8FmoE/VnDeVODBzHwoM18BrgdO7lwhM+/MzBfLh/eUn5uIGAfskpm3leu90KmeJKlA7p0oSVL9qyQRHJ2ZXwD+lJnXAicAEyo4bwTwWKfjtnLZtnwMuKV8/y+A9RFxU0SsjIjLyz2MkqSCuXeiJEn1r5LFYjaWb9dHxHjg98BbKzgvuinLbsqIiA8CLcCMTnG9G5gMPArcAMwGvt7lvHMpL1wzcuTICkJqDM7tkdQb+w9ror2bpM+9EyVJqh+V9AgujIi9KK0augRYA/zvCs5ro7TQS4dm4PGulSLiPcDngVmZ+XKnc1eWh5W+CiwG3tH13MxcmJktmdkyfPjwCkKqfx1ze9rXbyB5bW7P4pXt1Q5NUo1w70RJkurfdhPBiHgD8N+Z+Vxm3pWZB2XmmzPzaxU893JgTESMiohdKa08uqTL808GvkYpCXyyy7l7RURHdncUpQRUO+DcHkm95d6JkiTVv+0ODc3MzRFxPrCop0+cma+Wz11KafuIb2Tm/RExH2jNzCXA5cCewPciAuDRzJyVmZsi4v8Fbo/SAyuAf+lpDI3IuT2S+oJ7J0qSVN8qmSN4WzkpuwH4U0dhZj67oxMz80fAj7qUzet0/z3bOfc2YGIF8akT5/aonjn/VZIkqW9UMkfwo8B5wF2UeuZWAK1FBqWd59we1Svnv0qSJPWdHfYIZuao/ghEfaOjd6QqvSarF8Ht8+H5NhjaDEfPg4lnFP+6agjbm/9qr6AkSVLP7DARjIhzuivPzG/1fTjqC1WZ27N6EfzgAthYHpb6/GOlYzAZVJ9w/qskSVLfqWRo6JROP+8GLgZmFRiTatHt819LAjts3FAql/rAtua5Ov9VkiSp5yoZGvpXnY8jYijwr4VFpNr0fFvPyqUemjNzLBfddN9Ww0Od/ypJkrRzKlk1tKsXgTF9HYhq3NDm0nDQ7sqlPlDV+a+SJEl1ppI5gj8Asnz4BmAcO7GvoOrc0fO2niMIMLipVC71Efe2kyRJ6huV9Ahe0en+q8DvMtPxftpax4IwrhoqSZIkDXiVJIKPAk9k5ksAEdEUEW/NzEcKjUy1Z+IZJn6SJElSDahk1dDvAZs7HW8ql0mSJEmSalAlieAumflKx0H5/q7FhSRJkiRJKlIlieBTEbFl38CIOBl4uriQJEmSJElFqmSO4KeA70TEVeXjNuCc4kKSJEmSJBWpkg3lfwu8MyL2BCIz/1h8WJIkSZKkouxwaGhE/K+IGJaZL2TmHyNir4j4+/4ITpIkSZLU9yqZI3h8Zq7vOMjM54D3FheSJEmSJKlIlSSCgyJit46DiGgCdttOfVXb6kXwD+Ph4mGl29WLqh2RJEmSpAGkksVivg3cHhHfLB9/BLi2uJDUK6sXwQ8ugI0bSsfPP1Y6Bjd7lyRJkgRU0COYmZcBfw+8DRgH3AocWHBc2lm3z38tCeywcUOpXJIkSZKobGgowO+BzcD7gKOBtZWcFBHHRcS6iHgwIuZ28/jnImJNRKyOiNsj4sAuj/9ZRLR32rpCO/J8W8/KJUmSJDWcbSaCEfEXETEvItYCVwGPUdo+4sjM3GFiFhGDgKuB4yn1JJ4dEeO6VFsJtGTmROBG4LIuj/9P4KcVvxvB0OaelUuSJElqONvrEfw1pd6/kzLz8Mz8R2BTD557KvBgZj6Uma8A1wMnd66QmXdm5ovlw3uALdlKRPw/wL7Aj3vwmjp6Hgxu2rpscFOpXJIkSZLYfiL4PkpDQu+MiH+JiKOB6MFzj6DUi9ihrVy2LR8DbgGIiDcAXwLm9OD1BKUFYU66EoYeAETp9qQrXShGkiRJ0hbbXDU0M28Gbo6IPYBTgP8B7BsR/wzcnJk76qnrLmnMbitGfBBoAWaUiz4D/CgzH4vYdu4ZEecC5wKMHDlyB+E0kIlnmPhJkiRJ2qZKVg39U2Z+JzNPpDR0cxXwuoVfutEGHNDpuBl4vGuliHgP8HlgVma+XC4+DDg/Ih4BrgDOiYgF3cS2MDNbMrNl+PDhFYQkSZIkSapkH8EtMvNZ4Gvlnx1ZDoyJiFFAO3AW8JedK0TE5PJzHZeZT3Z6nQ90qjOb0oIylSSfkiRJkqQdqHT7iB7LzFeB84GllLabWJSZ90fE/IiYVa52ObAn8L2IWBURS4qKR5IkSZJUEpndTturOS0tLdna2lrtMCRJkiSpKiJiRWa2VFK3sB5BSZIkSdLAZCIoSZIkSQ3GRFCSJEmSGoyJoCRJkiQ1GBNBSZIkSWowJoKSJEmS1GBMBCVJkiSpwZgISpIkSVKDMRGUJEmSpAZjIihJkiRJDcZEUJIkSZIajImgJEmSJDUYE0FJkiRJajAmgpIkSZLUYEwEJUmSJKnBmAhKkiRJUoMxEZQkSZKkBmMiKEmSJEkNxkRQkiRJkhpMoYlgRBwXEesi4sGImNvN45+LiDURsToibo+IA8vlkyLiZxFxf/mxM4uMU5IkSZIaSWGJYEQMAq4GjgfGAWdHxLgu1VYCLZk5EbgRuKxc/iJwTma+HTgO+EpEDCsqVkmSJElqJEX2CE4FHszMhzLzFeB64OTOFTLzzsx8sXx4D9BcLv9NZj5Qvv848CQwvMBYJUmSJKlhFJkIjgAe63TcVi7blo8Bt3QtjIipwK7Ab/s0OkmSJElqULsU+NzRTVl2WzHig0ALMKNL+X7AvwIfzszN3Zx3LnAuwMiRI3sbryRJkiQ1hCJ7BNuAAzodNwOPd60UEe8BPg/MysyXO5X/GfBD4G8z857uXiAzF2ZmS2a2DB/uyFFJkiRJqkSRieByYExEjIqIXYGzgCWdK0TEZOBrlJLAJzuV7wrcDHwrM79XYIySJEmS1HAKSwQz81XgfGApsBZYlJn3R8T8iJhVrnY5sCfwvYhYFREdieIZwHRgdrl8VURMKipWSZIkSWokkdnttL2a09LSkq2trdUOQ5IkSZKqIiJWZGZLJXUL3VBekiRJkjTwmAhKkiRJUoMxEZQkSZKkBmMiKEmSJEkNxkRQkiRJkhqMiaAkSZIkNRgTQUmSJElqMCaCkiRJktRgTAQlSZIkqcGYCEqSJElSgzERlCRJkqQGYyIoSZIkSQ3GRFCSJEmSGoyJoCRJkiQ1GBNBSZIkSWowJoKSJEmS1GBMBCVJkiSpwZgISpIkSVKDKTQRjIjjImJdRDwYEXO7efxzEbEmIlZHxO0RcWCnxz4cEQ+Ufz5cZJySJEmS1EgKSwQjYhBwNXA8MA44OyLGdam2EmjJzInAjcBl5XP3Bv4OOBSYCvxdROxVVKySJEmS1EiK7BGcCjyYmQ9l5ivA9cDJnStk5p2Z+WL58B6guXx/JnBbZj6bmc8BtwHHFRirJEmSJDWMIhPBEcBjnY7bymXb8jHglp08V5IkSZJUoV0KfO7opiy7rRjxQaAFmNGTcyPiXOBcgJEjR+5clJIkSZLUYIrsEWwDDuh03Aw83rVSRLwH+DwwKzNf7sm5mbkwM1sys2X48OF9FrgkSZIk1bMiE8HlwJiIGBURuwJnAUs6V4iIycDXKCWBT3Z6aClwbETsVV4k5thymSRJkiSplwobGpqZr0bE+ZQSuEHANzLz/oiYD7Rm5hLgcmBP4HsRAfBoZs7KzGcj4n9SSiYB5mfms0XFKkmSJEmNJDK7nbZXc1paWrK1tbXaYUiSJElSVUTEisxsqaRuoRvKS5IkSZIGHhNBSZIkSWowJoKSJEmS1GBMBCVJkiSpwZgISpIkSVKDMRGUJEmSpAZjIihJkiRJDcZEUJIkSZIajImgJEmSJDUYE0FJkiRJajAmgpIkSZLUYEwEJUmSJKnBmAhKkiRJUoMxEZQkSZKkBrNLtQOoV4tXtnP50nU8vn4D+w9rYs7MsZwyeUS1w5IkSZIkE8EiLF7ZzkU33ceGjZsAaF+/gYtuug/AZFCSJDWsjRs30tbWxksvvVTtUKSaNmTIEJqbmxk8ePBOP4eJYAEuX7puSxLYYcPGTVy+dJ2JoCRJalhtbW288Y1v5K1vfSsRUe1wpJqUmTzzzDO0tbUxatSonX4e5wgW4PH1G3pULkmS1Aheeukl9tlnH5NAqRcign322afXPesmggXYf1hTj8olSZIahUmg1Ht98e+o0EQwIo6LiHUR8WBEzO3m8ekR8YuIeDUiTu/y2GURcX9ErI2IK6OGrhpzZo6lafCgrcqaBg9izsyxVYpIkiRJAB/96Ed585vfzPjx47dZJzO54IILGD16NBMnTuQXv/jFlseuvfZaxowZw5gxY7j22mt3+Ho/+clPOPHEE/sk9p1xzTXX8Pjjj1ft9fvarbfeytixYxk9ejQLFizots6Xv/xlxo0bx8SJEzn66KP53e9+t+WxQYMGMWnSJCZNmsSsWbN2+HqPPPLIdttK0RYvXsyaNWsKee7CEsGIGARcDRwPjAPOjohxXao9CswGvtvl3HcB04CJwHhgCjCjqFj72imTR3DpaRMYMayJAEYMa+LS0yY4P1CSJKkHFq9sZ9qCOxg194dMW3AHi1e29/o5Z8+eza233rrdOrfccgsPPPAADzzwAAsXLuTTn/40AM8++yyXXHIJ9957Lz//+c+55JJLeO6553YqjldffXWnzuupqiaCqxfBP4yHi4eVblcv6tXTbdq0ifPOO49bbrmFNWvWcN1113WbJE2ePJnW1lZWr17N6aefzt/8zd9seaypqYlVq1axatUqlixZstOx9NfnV5OJIDAVeDAzH8rMV4DrgZM7V8jMRzJzNbC5y7kJDAF2BXYDBgN/KDDWPnfK5BHcPfcoHl5wAnfPPcokUJIkqQc6VmFvX7+B5LVV2HubDE6fPp299957u3W+//3vc8455xARvPOd72T9+vU88cQTLF26lGOOOYa9996bvfbai2OOOabbpPLWW2/l4IMP5vDDD+emm27aUn7xxRdz7rnncuyxx3LOOefw0ksv8ZGPfIQJEyYwefJk7rzzTqCUvJ188skcd9xxjB07lksuuWTLc3z5y19m/PjxjB8/nq985SvA63utrrjiCi6++GJuvPFGWltb+cAHPsCkSZPYsKEf16tYvQh+cAE8/xiQpdsfXNCrZPDnP/85o0eP5qCDDmLXXXflrLPO4vvf//7r6h155JHsvvvuALzzne+kra2tR6+zYsUKDjnkEA477DCuvvrqLeXXXHMN73//+znppJM49thjyUzmzJnD+PHjmTBhAjfccANQ6gWePn06p556KuPGjeNTn/oUmzeX0p3rrruOCRMmMH78eC688MItz73nnntuuX/jjTcye/Zs/uu//oslS5YwZ84cJk2axG9/+9sevY8dKXLV0BHAY52O24BDKzkxM38WEXcCTwABXJWZa/s+REmSJA1E1VyFvb29nQMOOGDLcXNzM+3t7dss7+yll17iE5/4BHfccQejR4/mzDPP3OrxFStWsGzZMpqamvjSl74EwH333cevf/1rjj32WH7zm98ApaTnV7/6FbvvvjtTpkzhhBNOICL45je/yb333ktmcuihhzJjxgz22muvbt/H6aefzlVXXcUVV1xBS0tLn/xuKnb7fNjYJfHcuKFUPvGMnXrK7n7/995773bP+frXv87xxx+/5fill16ipaWFXXbZhblz53LKKae87pyPfOQj/OM//iMzZsxgzpw5Wz32s5/9jNWrV7P33nvzb//2b6xatYpf/vKXPP3000yZMoXp06cDpc9vzZo1HHjggRx33HHcdNNNvOtd7+LCCy9kxYoV7LXXXhx77LEsXry42xgA3vWudzFr1ixOPPFETj/99G7r9EaRPYLdzenLik6MGA28DWimlFAeFRHTu6l3bkS0RkTrU0891atgJUmSNHBUcxX2zNf/yRoR2yzv7Ne//jWjRo1izJgxRAQf/OAHt3p81qxZNDWVFhBctmwZH/rQhwA4+OCDOfDAA7ckgscccwz77LMPTU1NnHbaaSxbtoxly5Zx6qmnsscee7Dnnnty2mmn8Z//+Z998p773PPb6IXbVnkFKvn9d/btb3+b1tbWrZK5Rx99lNbWVr773e/y2c9+9nW9bM8//zzr169nxozSrLSOz6dDR48wlD6/s88+m0GDBrHvvvsyY8YMli9fDsDUqVM56KCDGDRoEGeffTbLli1j+fLlHHHEEQwfPpxddtmFD3zgA9x1110798voA0Umgm3AAZ2Om4FKByifCtyTmS9k5gvALcA7u1bKzIWZ2ZKZLcOHD+91wH2qj8dES5IkNZJqrsLe3NzMY4+9NrCtra2N/ffff5vlXW0vOdljjz223O8usdnWc2wrEQXYZZddtgw9BHq9rUCfGNrcs/IKVPr7B/iP//gPvvjFL7JkyRJ22223LeUd9Q866CCOOOIIVq5cudV5mdnvn1/X+v31+RWZCC4HxkTEqIjYFTgLqHRG5qPAjIjYJSIGU1oopnaGhhYwJlqSJKmRVHMV9lmzZvGtb32LzOSee+5h6NCh7LfffsycOZMf//jHPPfcczz33HP8+Mc/ZubMmVude/DBB/Pwww9v6Wm67rrrtvk606dP5zvf+Q4Av/nNb3j00UcZO7b0/m677TaeffZZNmzYwOLFi5k2bRrTp09n8eLFvPjii/zpT3/i5ptv5t3vfjf77rsvTz75JM888wwvv/wy//7v/77lNd74xjfyxz/+sa9/RTt29DwY3CVpH9xUKt9JU6ZM4YEHHuDhhx/mlVde4frrr+925c+VK1fyyU9+kiVLlvDmN795S/lzzz3Hyy+/DMDTTz/N3XffzbhxW69lOWzYMIYOHcqyZcsAtnw+3Zk+fTo33HADmzZt4qmnnuKuu+5i6tSpQGlo6MMPP8zmzZu54YYbOPzwwzn00EP56U9/ytNPP82mTZu47rrrtvQ87rvvvqxdu5bNmzdz8803b3mNIj+/whLBzHwVOB9YSimJW5SZ90fE/IiYBRARUyKiDXg/8LWIuL98+o3Ab4H7gF8Cv8zMHxQVa5/b3phoSZIk7VBRq7CfffbZHHbYYaxbt47m5ma+/vWvA/DVr36Vr371qwC8973v5aCDDmL06NF84hOf4J/+6Z8A2HvvvfnCF77AlClTmDJlCvPmzXvdwjNDhgxh4cKFnHDCCRx++OEceOCB24zlM5/5DJs2bWLChAmceeaZXHPNNVt6rw4//HA+9KEPMWnSJN73vvfR0tLCO97xDmbPns3UqVM59NBD+fjHP87kyZMZPHgw8+bN49BDD+XEE0/k4IMP3vIas2fP5lOf+lT/LxYz8Qw46UoYegAQpduTrtzp+YFQ6vm86qqrmDlzJm9729s444wzePvb3w7AvHnztqwCOmfOHF544QXe//73b7VNxNq1a2lpaeGQQw7hyCOPZO7cua9LBAG++c1vct5553HYYYdtGcbbnVNPPZWJEydyyCGHcNRRR3HZZZfxlre8BYDDDjuMuXPnMn78eEaNGsWpp57Kfvvtx6WXXsqRRx7JIYccwjve8Q5OPrm0luaCBQs48cQTOeqoo9hvv/22vMZZZ53F5ZdfzuTJk/t8sZjYXhdlLWlpacnW1tZqh1Fy8TC6nw4ZcPH6/o5GkiRpQFi7di1ve9vbqh3GgHfNNdfQ2trKVVddVe1QtBN+8pOfcMUVV2zVM1uE7v49RcSKzKxoZaBCN5RvWAWMiZYkSZKkvmIiWIQCabfM3wAABdJJREFUxkRLkiSpMcyePdvewBp2xBFHFN4b2BdMBItQwJhoSZIkSeorRW4o39gmnmHiJ0mS1MWOlueXtGN9sc6LPYKSJEnqF0OGDOGZZ57pkz9ipUaVmTzzzDMMGTKkV89jj6AkSZL6RXNzM21tbTz11FPVDkWqaUOGDKG5uXcLUZoISpIkqV8MHjyYUaNGVTsMSTg0VJIkSZIajomgJEmSJDUYE0FJkiRJajBRL6s2RcRTwO+qHUc33gQ8Xe0gVNdsYyqS7UtFsn2pSLYvFWmgtq8DM3N4JRXrJhEcqCKiNTNbqh2H6pdtTEWyfalIti8VyfalItVD+3JoqCRJkiQ1GBNBSZIkSWowJoLFW1jtAFT3bGMqku1LRbJ9qUi2LxWp5tuXcwQlSZIkqcHYIyhJkiRJDcZEsEARcVxErIuIByNibrXjUX2JiEci4r6IWBURrdWOR7UvIr4REU9GxK86le0dEbdFxAPl272qGaNq1zba18UR0V6+jq2KiPdWM0bVrog4ICLujIi1EXF/RPx1udxrmHptO+2rpq9hDg0tSEQMAn4DHAO0AcuBszNzTVUDU92IiEeAlswciHvYqAZFxHTgBeBbmTm+XHYZ8GxmLij/h9ZemXlhNeNUbdpG+7oYeCEzr6hmbKp9EbEfsF9m/iIi3gisAE4BZuM1TL20nfZ1BjV8DbNHsDhTgQcz86HMfAW4Hji5yjFJ0jZl5l3As12KTwauLd+/ltIXn9Rj22hfUp/IzCcy8xfl+38E1gIj8BqmPrCd9lXTTASLMwJ4rNNxG3XQYDSgJPDjiFgREedWOxjVrX0z8wkofRECb65yPKo/50fE6vLQUYftqdci4q3AZOBevIapj3VpX1DD1zATweJEN2WOw1VfmpaZ7wCOB84rD7uSpFryz8CfA5OAJ4AvVTcc1bqI2BP4N+Czmfnf1Y5H9aWb9lXT1zATweK0AQd0Om4GHq9SLKpDmfl4+fZJ4GZKw5GlvvaH8tyIjjkST1Y5HtWRzPxDZm7KzM3Av+B1TL0QEYMp/ZH+ncy8qVzsNUx9orv2VevXMBPB4iwHxkTEqIjYFTgLWFLlmFQnImKP8mRlImIP4FjgV9s/S9opS4APl+9/GPh+FWNRnen4A73sVLyOaSdFRABfB9Zm5pc7PeQ1TL22rfZV69cwVw0tUHkJ2a8Ag4BvZOYXqxyS6kREHESpFxBgF+C7ti/1VkRcBxwBvAn4A/B3wGJgETASeBR4f2a64Id6bBvt6whKQ6oSeAT4ZMd8LqknIuJw4D+B+4DN5eL/j9I8Lq9h6pXttK+zqeFrmImgJEmSJDUYh4ZKkiRJUoMxEZQkSZKkBmMiKEmSJEkNxkRQkiRJkhqMiaAkSZIkNRgTQUmSuoiITRGxqtPP3D587rdGRE3tNSVJqj+7VDsASZIGoA2ZOanaQUiSVBR7BCVJqlBEPBIR/zsifl7+GV0uPzAibo+I1eXbkeXyfSPi5oj4ZfnnXeWnGhQR/xIR90fEjyOiqWpvSpLUkEwEJUl6vaYuQ0PP7PTYf2fmVOAq4CvlsquAb2XmROA7wJXl8iuBn2bmIcA7gPvL5WOAqzPz7cB64H0Fvx9JkrYSmVntGCRJGlAi4oXM3LOb8keAozLzoYgYDPw+M/eJiKeB/TJzY7n8icx8U0Q8BTRn5sudnuOtwG2ZOaZ8fCEwODP/vvh3JklSiT2CkiT1TG7j/rbqdOflTvc34Zx9SVI/MxGUJKlnzux0+7Py/f8Czirf/wCwrHz/duDTABExKCL+rL+ClCRpe/wfSEmSXq8pIlZ1Or41Mzu2kNgtIu6l9J+pZ5fLLgC+ERFzgKeAj5TL/xpYGBEfo9Tz92ngicKjlyRpB5wjKElShcpzBFsy8+lqxyJJUm84NFSSJEmSGow9gpIkSZLUYOwRlCRJkqQGYyIoSZIkSQ3GRFCSJEmSGoyJoCRJkiQ1GBNBSZIkSWowJoKSJEmS1GD+L40wz9AeTffxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x1080 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot train and validation accuracies of the two models\n",
    "\n",
    "train_accs = []\n",
    "val_accs = []\n",
    "for dropout in dropout_choices:\n",
    "    solver = solvers[dropout]\n",
    "    train_accs.append(solver.train_acc_history[-1])\n",
    "    val_accs.append(solver.val_acc_history[-1])\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].train_acc_history,\n",
    "             'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.subplot(3, 1, 2)\n",
    "for dropout in dropout_choices:\n",
    "    plt.plot(solvers[dropout].val_acc_history,\n",
    "             'o', label='%.2f dropout' % dropout)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(15, 15)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 2:\n",
    "Compare the validation and training accuracies with and without dropout -- what do your results suggest about dropout as a regularizer?\n",
    "\n",
    "比较有和没有dropout的验证和训练的准确性——你的结果对dropout作为一个正则化方法有什么建议?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "pdf-inline"
    ]
   },
   "source": [
    "## Inline Question 3:\n",
    "Suppose we are training a deep fully-connected network for image classification, with dropout after hidden layers (parameterized by keep probability p). \n",
    "\n",
    "If we are concerned about overfitting, how should we modify p (if at all) when we decide to decrease the size of the hidden layers (that is, the number of nodes in each layer)?\n",
    "\n",
    "假设我们正在训练一个深度全连接网络来进行图像分类，隐层(保持概率为 p 的参数化)后会有dropout。\n",
    "\n",
    "如果我们担心过拟合，那么当我们决定减少隐藏层的大小(即每个层的节点数)时，应该如何修改p(如果有的话)?\n",
    "\n",
    "## Answer:\n",
    "[FILL THIS IN]\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
